This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Greptile Summary

This PR adds a comprehensive Fern tutorial (audio-pretrain.mdx) that documents the seven-stage long-form audio cutting pipeline for ALM pretraining, along with cross-links from the audio landing page, tutorial index, ALM tutorial, data builder, and overlap-filtering reference.

• New tutorial page: Covers the full workflow from input manifest schema and path-resolution modes through dry-run preview, full extraction, and Python API usage, with accurate documentation of CLI defaults, stage contracts, greedy packing logic, repetition filtering, and shard lifecycle.
• Navigation additions: fern/versions/main.yml, curate-audio/index.mdx, and curate-audio/tutorials/index.mdx receive new card/entry entries; alm.mdx, data-builder.mdx, and overlap-filtering.mdx receive cross-link bullets.
• Accuracy verified: Stage contracts, CLI argument defaults, Python API call signatures (including prepare_audio_pretrain_outputs / finalize_audio_pretrain_outputs parameter order and audio_filepath_key), histogram boundary semantics, and corrupted_audio/missing_audio sparsity behavior all match the implementation in planning.py, io.py, finalize.py, and run.py.

Confidence Score: 5/5

Documentation-only change with no modifications to runtime code; all technical claims verified against the implementation.

Every stage contract, CLI default, Python API call signature, and metrics field name in the new tutorial was cross-checked against planning.py, io.py, finalize.py, utils.py, and run.py. The histogram boundary note, shard lifecycle warnings, dry-run stale-tar caveat, and corrupted_audio/missing_audio sparsity behaviour all match the source. Navigation entries in main.yml and the cross-link bullets in the five existing pages are correct. No runtime code is touched.

No files require special attention.

Important Files Changed

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Long-form JSONL manifest"] --> B["Stage 0: ReadLongFormManifestStage\n(fan-out: 1 task per row)"]
    B --> C["Stage 1: OverlapFilterStage\n(drop empty + overlapping segments)"]
    C --> D["Stage 2: SnippetCutPlannerStage\n(greedy pack into bounded snippets)"]
    D --> E["Stage 3: SnippetRepetitionFilterStage\n(n-gram repetition drop; no audio I/O)"]
    E --> F["Stage 4: SnippetExtractionStage\n(fan-out: 1 task per snippet;\nread to mono to resample to encode to tar shard)"]
    F --> G["Stage 5: SnippetManifestWriterStage\n(per-worker JSONL shards)"]
    G --> H["Stage 6: PretrainMetricsAggregatorStage\n(per-worker metrics JSONL shards)"]

    subgraph Driver ["Driver-side finalization"]
        I["prepare: delete stale shard files"]
        J["finalize: merge manifest + metrics + tar shards"]
        K["reconcile manifest against tar"]
        L["patch metrics with reconcile drop counts"]
        I --> J --> K --> L
    end

    B -.->|"dry-run: skips audio I/O"| F
    H --> Driver

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["Long-form JSONL manifest"] --> B["Stage 0: ReadLongFormManifestStage\n(fan-out: 1 task per row)"]
    B --> C["Stage 1: OverlapFilterStage\n(drop empty + overlapping segments)"]
    C --> D["Stage 2: SnippetCutPlannerStage\n(greedy pack into bounded snippets)"]
    D --> E["Stage 3: SnippetRepetitionFilterStage\n(n-gram repetition drop; no audio I/O)"]
    E --> F["Stage 4: SnippetExtractionStage\n(fan-out: 1 task per snippet;\nread to mono to resample to encode to tar shard)"]
    F --> G["Stage 5: SnippetManifestWriterStage\n(per-worker JSONL shards)"]
    G --> H["Stage 6: PretrainMetricsAggregatorStage\n(per-worker metrics JSONL shards)"]

    subgraph Driver ["Driver-side finalization"]
        I["prepare: delete stale shard files"]
        J["finalize: merge manifest + metrics + tar shards"]
        K["reconcile manifest against tar"]
        L["patch metrics with reconcile drop counts"]
        I --> J --> K --> L
    end

    B -.->|"dry-run: skips audio I/O"| F
    H --> Driver

_{Reviews (9): Last reviewed commit: "Merge branch 'main' into codex/docs-long..." | Re-trigger Greptile}

🌿 Preview your docs: https://nvidia-preview-codex-docs-long-form-audio-cutting.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-audio
• nemo/curator/curate-audio/process-data/alm/data-builder
• nemo/curator/curate-audio/process-data/alm/overlap-filtering
• nemo/curator/curate-audio/tutorials/alm
• nemo/curator/main/curate-audio/tutorials/audio-pretrain
• nemo/curator/curate-audio/tutorials

@sarahyurick @yqwangustc @mohammadaaftabv friendly ping on this docs PR for the #2118 release-docs workstream — when you have a moment, could you take a look at the long-form audio cutting guide?

Docs preview: https://nvidia-preview-codex-docs-long-form-audio-cutting.docs.buildwithfern.com/nemo/curator

Thanks!

Thanks for the detailed review. This PR is docs-only (we handle documentation, not the stage code), so I've updated the guide to accurately document the current behavior and added the "at minimum" guidance you called out. The underlying behavior changes are flagged below as follow-up engineering work.

Fixed in docs (commit b20eb2c)

• ID collision after normalization — The id field now states IDs must be unique after tar-key normalization, and explains that make_snippet_id replaces ., /, and \ with _, so session.1 and session_1 collide into the same tar member name.
• Missing segments key — Marked segments as Required: a missing key fails OverlapFilterStage input validation (can fail the pipeline), while an empty list [] is accepted and yields zero snippets. Applied your suggested wording.
• Sparse reconciliation counters — Removed the misleading "corrupted_audio": 0 from the example metrics JSON, documented that missing_audio/corrupted_audio are written only when nonzero, and recommended defensive access (dropped.get("corrupted_audio", 0)) to avoid KeyError.
• Dry-run vs. stale tar — Added a warning requiring a fresh, nonexistent --output-audio-tar-path for every dry run, explaining that finalization otherwise reconciles the preview manifest against a stale archive (silent row removal / validation against unrelated audio).

The greptile Python-API audio_filepath_key and histogram 30–60 bin notes were already addressed in the prior commit.

Suggested engineering follow-ups (out of scope here — code, not docs)

These change stage behavior and belong in a code PR against nemo_curator/stages/audio/alm/pretrain/:

1. Dry-run-aware finalization — Have finalize_audio_pretrain_outputs merge manifest/metrics shards but skip tar merge + _reconcile_manifest_with_tar in dry-run mode (or reject an existing tar path in dry-run). Add a regression test for a dry run against a pre-existing tar.
2. Uniqueness after normalization — Validate ID uniqueness after applying the same make_snippet_id normalization (or use a collision-free escaping/hash scheme). Add regression coverage for session.1 vs session_1 and path-separator variants.
3. Reader-side segments validation — Reject rows with a missing segments key in ReadLongFormManifestStage with their JSONL line number, rather than deferring the failure to OverlapFilterStage.
4. Always-initialize counters (optional) — Initialize missing_audio/corrupted_audio to 0 in the runtime schema and update the tests, if a dense schema is preferred over the current sparse one.

Happy to open a tracking issue for these if that's useful.

ok to test b20eb2c

ok to test dcfcc1f

@mohammadaaftabv thanks again for the detailed review — this is now ready for another pass. Since the PR is docs-only, I updated the guide to accurately describe the current behavior and added the "at minimum" guidance you flagged (commit dcfcc1f). Your five inline points (dry-run vs. stale tar reconciliation, ID collision after make_snippet_id() normalization, missing segments validation, and the sparse reconciliation counters) are all addressed in the docs, with the underlying stage-behavior items called out as follow-up engineering work. Could you re-review and, if it looks good, clear the change request? @yqwangustc has already approved.